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Abstract 

The measurements with the background estimation from an off-zone are widely used in as- 
trophysics and accelerator physics. In this note an overview of the statistical methods which 
estimate the range and the significance of the measured signal is done. The method which in- 
cludes a systematic uncertainty is developed for the on/off-zone measurements and compared 
with other existing methods. 

1 Introduction 

The main goal of any signal measurement is to estimate the signal value and the precision of the estimation. 
For this purpose the most probable value of the signal and also the the signal range are provided as a results 
of the measurement. The probability of the signal to be between the range limits is fixed to some value 
(90% for example or 1 — a). In terms of statistics this mean that measurement X was done and one wants 
to provide the value of the signal s which maximises the probability P(s | X) and also the limits s m m, s max 
for which P{s m i n < s < s max ) = 1 — a. The values s, s rn i n , s max can be calculated if P(s \ X) is known for 
every s. 

Actually in the most cases it is straightforward to know the probability P(X \ s) of observing X if the 
signal s is known. This can be done assuming Poisson or another model involved in the measurements. To 
obtain P{s \ X) one can use the Bayes' theorem: 

P(X | s)P(s) 

P{S 1 X) = P(X) (1) 

For P(s | X) the denominator is simply a constant as it depends only on the measurement X, so it can 
be calculated using the probability property /_ °° P(s \ X) As = 1. The remaining multiplier P(s) is the 
unknown probability distribution of the signal. This formula shows that strictly saying the measurement X 
does not give the answer on how much could be the signal. To go beyond this paradox two approaches were 
developed by the statisticians. As the "true limits" cannot be obtained it is expected at least to have some 
reasonable estimations which should follow naive expectations such as: 

• The upper limit is smaller if the measured value is smaller in the measurements which arc done with 
the same background. 

• The limits are broader if the background is higher in the measurements with the equally estimated 
value of the signal s. 

• The upper limit is lower if the systematic errors are smaller in the measurements with the same 
background and the same measured X . 
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To find P(s | X) from the formula ([T]) one can use a Ptheo(s) which is based on some belief, physical 
limits or previous measurements. This approach is called Bayesian. The simplest estimation of Ptheo(s) 
may be realised by setting a uniform distribution between the physical limits. However it is known that 
the limits arc different for the different Ptheo(s)- P(s \ X) obtained from the formula (TTJ) using Ptheo{s) is 
named a post probability. 

An another approach is used by the frcqucntists. In this approach one sets limits in a way to have a true 
value of the signal inside the limits in 1 — a part of the all measurements. This is also called a coverage of 
the method. The advantage on this approach is that it is based only on P(X \ s) or, equivalently, on the 
likelihood L(s \ X) = P(X | s£J 

A frequentist method developed by Neyman consists of the two steps [Nevmanl . Il937 1 : 



• To find some "confidence belt" for every signal s. The confidence belt is a set of different measurements 
{X^} which give Y^i=i I -^T) — 1 — a - The way to choose the confidence belt {X^} is arbitrary. 

• To use a real measurement X. If X is present in some set of {X^} for the signal s then this signal s 
is included to the confidence area. This operation is done for every set {X^} to build the area. The 
upper limit with 1 — a confidence level is the maximum signal in the confidence area. 



Limits obtained by this method may vary depending on the choice of the confidence belts. In [Nevman 



1937j two options were suggested: central and one-sided limits (upper/lower). Both types have some un- 
wanted features. Central limits may be too strict for the case when the number of the observed events is 
lower than the expected number from the background. One-sided limits do not allow to claim a discovery or 
setting the upper limit depending on the measured X. The mentioned problems are essential as one should 
define which type of the limits one wants to set before the measurement. If this decision is done after the 
measurement this would lead to the under-coverage of the limits known as a "flip- flopping" problem. 

2 Case of the signal with the known background 

The methods described here are applied to the processes with a Poisson likelihood: 

s «obsg-S 

£(n-obs I s; b) = j— = Poisson(n obs | s + b) (2) 

^obs • 

where X = n b s and " ;" symbol is used to distinguish the unknown variables from the known ones which 
are for this case the signal and the background correspondingly. 

In ord er to avoid the " flip-floppin g" problem the Neyman's method was improved by the special ordering 



principle [Feldman fc Cousins! . |1998J . The advantage is that the procedure to define an upper limit in case 
of a low number of observed events or a two-sided limit in the opposite case is defined a priori. Confidence 
belt selection is based on sorting the {X n } by the decreasing rank: 

R = &{n obs | s;b) 
£(n obs | s;b) 

where £(n b s | s]b) is a maximum possible likelihood for the fixed number of observed events ri bs which is 
realised by the signal s. 

For some cases of the low number of the observed evens the measurements with a higher background may 
provide better limit with this method. Actually, this seeming discomfort is connected with the difference 
between Bayesian and frequentist approaches. In the latter, if a signal plus background is measured, then 
if the expected mean background is bigger the signal should be lower (in average over a lot of trials). But, 
on the another hand, it is expected that the experiment with bigger expected background can not provide 
better limits. 

Bayesian method consists of the following steps: 

• To choose Ptheo(s). In a simple case a uniform prior distribution may be chosen. In this case the post 
probability P(s) oc £(n b s | s;b). Some sophisticated approaches may have a more complicated prior, 
but the obtained limits are similar in the most cases. 



1 Plcase, note that in the following, the order of the observables is changed with the variables in the likelihood description, so 
L(X | s) = L(s | X). This is convenient to describe the likelihood with classical processes/distributions like Gaussian(x | x,a) 
and Poisson(x | A) where the variables (which are normally the observables and the nuisance parameters of an experiment) 
are separated from the parameters (which are normally the known or searched variables of an experiment). Integration of the 
likelihood/probability over all sets in the left part from the "|" should give unity. 
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Figure 1: Calculation of the upper limits with the Feldman & Cousins universal approach (solid line) and 
the Bayesian approach (dotted line) for the case of 90 expected background. 



• To find si owcr and s U ppcr using n OD s from the measurement. The integration of the likelihood till 1 — a 
confidence level is used: 

poo , ; — 7T-r- > 1 - a (4) 

J_ oo L{n obB I s;b)ds 

The integration can be done numerically using steps for s and ordering the likelihood corresponding 
to each step by the decreasing order. Once the ratio becomes equal or more than 1 — a the integration 
stops and the minimum and maximum values of s become the limits. 

Comparison between the limits obtained by the Feldman & Cousins universal approach and the Bayesian 
approach is presented in fig. [TJ The limits are very similar for the case when n b s > b and the frequentist 
upper limit is tighter when n b s < b. The latter is a known problem which was discussed a lot during last 
two decades. As a solution some mixed approaches were developed. 

One of the propo sed solutions was in a usage of a so-called conditional PDFs |Roe fc WoodroofeL Il999t 



Conrad et all l2003j . This PDF uses the current measurement n b s and the fact that the background 



can not be bigger than the n b s . The advantage is that for zero observed events upper limit is about 
2.42 independently from the background mean b. Instead, the universal approach provides upper limit 
2.44 for zero mean background, 1.08 for 3 mean background and so on decreasing, in contrast to the 
Neyman's upper limit which is 2.3 for zero background. Lately it was found that this method with a 
conditional PDFs suggests a presence of th e signal in all ca ses if the observed value is bigger than the mean 



background for the continuous observable [Cousins! . I2000l | . Also the method does not have a coverage in 
the frequentist sense. One of the proposed s olutions was to change the lower limit to have 1 — a coverage 
for each signal [Mandelkern fc Schultz , 2000j . Another solution, proposed by the authors of the conditional 



PDF themselves was t o use a Bayesian limits w ith the uniform distribution of the signal in the physically 
allowed range (>= 0) (Roe fc WoodroofeL l2000j . 



Actually, even acknowledging this discomfort with an upper limit decrease while increase of the back- 
ground in case of zero events Feldman & Cousins method is still a default method for the upper limits 
calculation among physicists. However in the form it is described in the former paper it does not allow to 
include systematic errors in the calculation. 

3 Including uncertainties to the known background 

For a measurement with a background with a known uncertainty b ± Cf, (statistic plus systematic error) the 
likelihood can be modified as: 

£(n bs, b' | s; b, at) = Poisson(n b s | s + 6')Gaussian(6' | 6, ab) (5) 
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Figure 2: Upper limi t calcu lation with the Bayesian method using the profile likelihood ([5]) and x 2 approx- 
imation |Rolke et al. . 2004l | (blue) compared to the calculation with the exact Bayesian method with the 
same profile likelihood (red) for the case of 90 ± 6 expected background. The upper limit with the Bayesian 
method for the case of the 90 expected background (2J is also shown (black). 



where b' is the mean background in the measurement n f, s . It is assumed that b' has a gaussian distribu- 
tion with a mean b and a sigma a b (however in some literature systematic errors are approximated with a 
uniform distribution) . Actually the presence of the distribution of b' in this likelihood referred to a Bayesian 
approach. So, the limits obtained by using this likelihood arc Bayesian or scmi-Bayesian depending on the 
further calculation procedure. 



Investigating the tools available in the ROOT framework the method described by Rolke et al [Rolke et al 



2004I ] was found. This method is a modified "£ + |" method which means that it is a maximum likelihood 



estimator (MLE) where the likelihood ratio is used to find the most probable signal and the background. 
Practically, MLE method is identical to the Bayesian approach with the uniform prior distribution of the 
signal and the profile likelihood L(n i, s ,b' \ s,b,at,). Here b 1 means that for every n i, s the best value of b' 
which maximises the likelihood is found. The obtained likelihood does not depend on b' anymore. 
As in the Bayesian approach limits si ower , s U p Pper according to ([T]) should satisfy the equation: 

r uppor £(n obs , &' | s; b, a b )P thco (s) ds 

— 1 °^r J > 1 — a (6) 

/-co £ ("obs, b' | s; b, cr b )P thco (s) ds 



To make calculations faster the theorem that — 21ogi? has approximately x 2 was used [Rolke et al. . 2004 1. 
Exact Bayesian profile likelihood method and the x 2 assumption is compared in fig. [5] for the case of 90 ± 6 
expected background. It was found that approximation with x 2 gives lower limits than the exact calculation. 
Bayesian limit for the background without uncertainty is also presented for the comparison. This limit is 
always lower than the limit with an uncertainty as it is expected. 

Another approach is a frequentist construction of the N eyman's confidence belts based on the so-called 
marginal likelihood which is a likelihood integrated over b' [Conrad et al. 



L(n ohs \ s;b,a b ) = L(n ohs ,b' | s;b,<r b ) db' (7) 
Jo 

The obtained likelihood is a function of the signal and the observed events only so the upper limit 
calculation may follow the Feldman & Cousins method. The obtained limit for 90 ± 6 expected background 
in comparison with the Bayesian profile likelihood limit calculation is present in fig. [3] Again one can see 
that the Bayesian limits are higher than t he frequentist ones w hen n b s < fe- 
lt is worth to note that this approach Conrad et al. I. l2003j has some unwanted behaviour for the mea- 



surement with the systematic uncertainty of the signal efficiency. The limits are becoming better while the 
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Figure 3: Calculation of the upper limit with the Bayesian method using the profile likelihood from ([5]) 
(red solid) , compared w i th th e upper limit calculated with the frequcntist approach using the marginal 
likelihood Conrad et al. |, |2003| (black solid) for the case of the 90 ± 6 expected background together with 
the Bayesian upper limit (red dotted) and the limit obtained with the Fcldman & Cousins approach (black 
dotted) for the case of the background 90 without a systematic error. 



uncertainty on the efficiency is increasing. The suggested correction was to change t he likeliho od in the 
denominator of the rank on the best likelihood given no uncertainty on the efficiency Hill 20Q.'i . For the 
problem with the uncertainty on the background the original method Conrad et al. . 2003| behaves correctly 
so no such correction is mandatory. 



4 On/off-zones measurements 



In many experiments it is possible to estimate the background from an additional measurement in the 
zone where no signal is expected. This so called off-zones approach is widely used in astrophysics and in 
accelerator physics. The selected off-zone can be an another part of the sky in the case of the astrophysic 
measurements, an another part of the energy spectrum or a measurement with the switched off accelerator 
in the case of the accelerator experiments. It is important to mention that the off-zone can have a bigger 
statistic, for example the area can be geometrically bigger or the exposure time can be longer. Ratio of the 
statistic in the off-zone to the on-zone is called r in following. So, if the number of the background events 
in the on-zone is expected to be b then in the off-zone the number of the background events is expected to 
be rb. 

During such measurements n bs and rib g in the on-zone and in the off-zone correspondingly are obtained. 
The number of the events n b s is distributed around s + b and rib g is distributed around rb. The choice of 
the distribution depends on the experiment. In most cases, a Poisson distribution may be used. So, rt b s 
and nbg are distributed as: 

P(n bs | s, b) = Poisson(n obs | b + s) (8) 



P(rib g | b; t) = Poisson(nb g \ rb) 
And if the two measurements are independent then: 

P(n Q bs, "-bg | s, b; t) — Poisson(n b s | b + s)Poisson(rib g | rb) 



(9) 



(10) 



This probability was used for the significance estimation in the gamma-ray astronomy [Li fc Mai . Il983j . 
Also it might be used for the limits estimation. It is important to note that n b s and rib g are treated in the 
formula in a similar way which corresponds to the same measurement procedure for these variables. This 
likelihood has two parameters s and b. 
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Sometimes, during the on/off-zone measurements size or efficiency of each zone can be estimated with 
a some limited precision resulting as a systematic error. This fact should be included in the probability 
distribution. It can be done with a two gaussian distributions. The likelihood distribution in this case 
becomes as follows: 

£("obs, n hg , a on , a oS \ s,b;r,a) = 

— Poisson(n, D bs | OL on b + s)Gaussian(a on | lj cr)x 

x Poisson(ribg | a ff?"&)Gaussian(aofT | 1, c) (11) 

where a is a systematic uncertainty. 

Analytical/numerical construction of the Neyman intervals with the likelihood ratio ranking is possible 
if the likelihood is first marginalised over a on and a g. The marginal likelihood becomes a function of s 
and b variables. The obtained confidence area will depend on the s and b and so the limit on s from this 
area will be quite conservative. Additionally, the integration over a on and a g can be done only numerically 
which makes the computation very heavy. 

The Bayesian approach is, instead, easily applicable to this likelihood. The fastest way is to use the 
profile likelihood which depends only from s. The post probability area of 1 — a confidence level size can 
be calculated as: 

j^uppcr £( nobg) nbgi a on a oB | s, b; r, a) ds 

/cT , — — l r—>l-« (12) 

J-oo M n °bs, "bg, aonttoff | s, b; r, a) as 

It was not succeeded to calculate the profile likelihood from (TlTj) analytically. However, from the three 
equations based on the fact that partial derivatives by b, a on , a g are equal to zero at the maximum of the 
likelihood one can obtain: 

a on (l - aw) + a ff(l - a g) = (13) 
(a on + a ofi T)b 2 - (n on + n oS — (a on + a Q sT))b —n oB = (14) 

C^on a n 

For a one chosen parameter a on , for example, other parameters, b and a g can be calculated by solving 
these quadratic equations. If one of the solutions is negative, then zero value should be used instead it. The 
maximum likelihood can be found by scanning a on in a range l±3cr numerically. The performed calculations 
with a 1% precision takes about ~ 1 sec to find a limit numerically for a single measurement using a 2GHz 
CPU. 

The profile likelihood of (TTU)) has b value which can be calculated similarly to (TT4"]) : 

(1 + r)b 2 - (n on + n off - s(l + r))b - n off s = (15) 



5 Comparison of the methods 

All these methods were applied to a particular astrophysical measurement for a test. In this measurement 
three similar off-zone regions with a size of the on-zone were selected on the sky. All four zones (one on-zone 
and three off-zones) have the same visibility. The number of the background events expected in the on-zone 
and the off-zones was estimated to be the same inside a systematic uncertainty of 3%. After the optimisation 
of the cuts nbg = 270 events in the three off-zones were seen. 

All Bayesian methods with a different likelihoods are compared in fig. 0] for this measurement. Both 
likelihoods with an introduced background uncertainty provide higher upper limits comparing with a Poisson 
processes without uncertainty. Likelihood with two Poissons for n D b s and n^ g (|11[) provides better limits 
and follows the shape of the upper limits in the case of the no uncertainty. 

The Bayesian approach using a profile likelihood with a two Poisson distributions and a Gaussian sys- 
tematic uncertainties (jlip is compared with a frequentist approach using a marginal likelihood with the 
background with a known uncertainty © in fig. [5] 



6 Signal flux estimation using simulations 

In the previous sections the signal was estimated as a number of events. However sometimes the result 
should be presented as some source quantity (flux in astrophysics measurements, cross-section in acceleration 
experiments for example). Simulation of the source provides conversion of the latter quantity to the number 
of the signal events using a simple proportion: 



(» 




Figure 4: Calculation of the upper limits with the Bayesian methods. A process with a known background ([2]) 
for b — 90 (black), using a profile likelihood for the process with a known background and a gaussian 
uncertainty ([5]) for b = 90, erf, = 6 (green), using a profile likelihood for a process with a two Poisson 
distributions (fTU|) for n;, g = 270, r = 3 (red) and using a profile likelihood for a process with a two Poisson 
distributions with Gaussian uncertainties (jTTj) for n b s = 270, r = 3 and a = 3% (blue). 




Figure 5: Calculation of the upper limits with the likelihood (J5]) for b = 90, = 6 with the Conrad's 
method (black) and with the likelihood (TTTI) for rib g = 270, r = 3 and a = 3% with the Bayesian method 
using a profile likelihood (red). 
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Simulation, however, is usually affected by a systematic uncertainty: 



£ (Xim) = Gaussian(sg im | s sim ,a sim ) (17) 
where the systematic uncertainty of the si mulation is a S un ■ Th e uncertainty as a ratio Usim/ssim is equivalent 



to the detection efficiency uncertainty in jConrad et all 12003 1 . 



The likelihood for / can be written using the equation for signal s: 

/ 



= s sim -r— (18) 

/sim 



and ([5]) in case of the classical measurements. The obtained likelihood is: 

£(n obs , b , s sim | /; b, / sim , a b , 



^sim ? w sim ) 



S • 

= Poisson(n obs | /-^ + b') Gaussian (6' | b, cr b )Gaussian(Sg im | s sim , cr sim ) (19) 

/sim 

or for the case of the off-zones measurements using ffTTI) : 

^(^obs: ^bg? t^on; ^off; ^sim I f i ^' y*sim; ^3 ^sim? ^sim) 

s >. 

— Poisson(n Q bs | ot on b + f sim )Gaussian(o; on | l,cr)x 

/sim 

x Poisson(nbg | a ffr6)Gaussian(a ff | l,cr)x 

x Gaussian(sg im | s sim ,cr sim ) (20) 

The likelihood (fH?)) allo ws a marginal i zation and a frequentists limits construction which can be followed 



with minimal changes by [Conrad et all l2003j . The Bayes approach using a profile likelihood is also ap- 
plicable. The likelihood ([20| should be used with a Bayes approach as it depends from the two unknown 
parameters s and b (sec discussions in section 2]). The equations of the first derivatives for the profile 
likelihood parameters: 

f_ _ 4m ~ ^im = Q (21) 




/sim 0*j 

( 



sim 



1 ra off = (22) 



^-obs 



a on b+ ff^ 



V ctosrb 

1 | & _«on_ 1 l =0 (23) 



/si 

( - 1 W **fi = (24) 

where equation fj 13[) can be obtained again from the last three equations. The parameters d on , a g, s' sim , b 
can be found by scanning a s and obtaining b from equation ([2"4"]l . d on from equation ([T3]) and Sg im from (|2"3"|) . 
Zero value should be used instead of the negative result for any of these parameters. The obtained limits 
calculated with both likelihoods are demonstrated in fig. [6] 



7 Significance estimation. 

To demonstrate the significance of the signal discovery the positive upper limit estimation may not be 
enough. Traditionally, an estimation how compatible is the measurement with the no signal hypothesis 
should be done also. If the process has a gaussian distribution with a mean b and a sigma a and n b s 
was measured than the significance can be estimated simply as S = (n bs ~ b)/<r and one can say that 
"S standard deviation was observed". For more complicated processes some test statistic which has a 
gaussian distribution for a background only hypothesis may be f ound. 

This approach for the on/off-zone measurements is shown in |Li fc Ma . 1983j . Basically it is done with 
a model hypothesis testing using a likelihood ratio test: 

A = £{n ohe ,n hg I s _ = ^5) 
£(n ob s,n.b g I s,b) 
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Figure 6: Calculation of the upper limits with the likelihood p^|) for b = 90, <Jb = 6, / s im = 1.2, s s im = 5.4, 
(Tgim = 20% with the Conrad's method (black) and with the likelihood (|2"0"|) for rib g = 270, r = 3 and a = 3%, 
/sim = 1-2, Ssim = 5.4, cr s i m = 20% with the Baycsian method using a profile likelihood (red). 



where the likelihood maximum for the no signal hypothesis is found for the numerator and the best likelihood 
for all hypotheses is foun d for the denomina t or. T he value of —2log\ has approximately x 2 distribution for 
the no signal true model (Casella fe BergerT 12001 . chap. 10.3.1]. Therefore, \/—2logX has approximately a 



gaussian distribution and may be used as a significance estimation. 

For the off-zones method with systematic errors the same procedure was applied. To find £(n b S )^bg | 
s = 0, b) or £(n b s , fibg | sb) a system of equations based on the fact that partial derivatives are zero at the 
local maximum was used. It was found for the denominator: 

b=— — ; s = n on - a on b; d on = 1; a g = 1; (26) 

0L o ffT 

and for the numerator one can use equations (|13|) and (|14[) assuming s=0 and adopting the same scan 
procedure described in the previous section. The first equation remains the same and the last one can be 
solved as: 

, , ton + n oS 

bi =0, b 2 = ■ (27) 

Oion + OioffT 



The obtained significance a s a \/—2logX wa s tested for a model with n^g = 270, r = 3, a = 3% and compared 
with the significance from [Li fc Ma . Il983j in fig. [7] using a Toy Monte Carlo simulations. One can see that 



the calculated significance has an almost gaussian distribution. 



8 Conclusions 

In this note an overview of the existing methods for setting the upper limits with a presence of the unknown 
(nuisance) parameters was done. The methods were used for the signal limits calculation for the case of 
the on/off-zones measurements. Also a new likelihood for this case of the measurements was constructed. 
It treats both observations in the on-zone and in the off-zone in the similar way and also includes the 
uncertainty of the expected background between the zones. 

It was shown on a particular example that all described methods with the profile and marginal likelihoods 
give the similar upper limits for the case when the number of the observed events is bigger than the 
expected background. If the number of the observed events is less than the background expectation then 
the frequentist methods tend to give smaller limits compared to the Bayesian methods. 

The suggested method for the limits calculation in case of the on/off-zone measurements is the Bayesian 
approach together with the two Poisson distributions - one for the on-zone and another for the off-zone. It was 
shown also how to include the systematic uncertainties in this case. The significance of the signal presence 
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Figure 7: Significance calculated for the Toy Monte Carlo simulations of the on/off-zone measurements with 
the mean background b = 90, r = 3 and the systematic uncertainty a = 3% for the model with an ab sence 
of the signal (black) compared to the calculations without a systematic uncertainty Li fc Ma , 1983| (red) 
and a gaussian distribution (smooth black line). 



using a background only hypothesis testing with a likelihood ratio test was presented for this likelihood as 
well. It was demonstrated that the significance distribution follows well the gaussian distribution with the 
mean and the sigma 1. 
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